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A sensor-referenced or sensor-guided manipulator control task contains a 
goal or a set of subgoals. The control goal or subgoals are expressed as a 
combination of various sensory data. The simultaneous occurrence of time 
trajectories of various sensory data at a single point or within a given su - 
volume of a multidimensional data space can be called a sensory information 
"event". Hence, sensory information "event" is the projection or mapping of 
the control goal or subgoals Into a multidimensional data space. 

Figure 1 gives simple Illustrations for the concept of sensory informa- 
tion "event". Equal length of two proximity sensor beams can be an event 
in the sense that it may signify, e.g., the roll, yaw or pitch alignment of 
a mechanical hand relative to an object. Equal magnitude of two orthogonal 
force components can be an "event" in the sense that it may signify, e.g., 
the push or pull of an object by a mechanical hand in a given direction. Or, 
for instance, half contact coverage of a touch-sensitive area on a mechanical 
finger can be an "event" in the sense that it may signify, e.g., that there 
is sufficient contact between hand and object for successful grasp. 

The operator's attention in both manual and computer control is normally 
focused at the control goals or subgoals, that is, at the sensory information 
"events". Typically, when such "events" occur, some control action must be 
taken. It is to the operator's advantage to have these sensory "events dis- 
played in easily perceivable and unmistakably unique forms. In the absence 
of such "event" displays, the operator must determine the occurrence of the 
"event" by following and evaluating a multidimensional set of data in real 
time. This is not only a demanding task and heavy workload for the operator, 
but also a common source of errors . 


III. DISPLAY OF "EVENTS" 


Event-driven displays can be implemented by developing and/or employing 
appropriate real-time algorithms which (a) coordinate and evaluate sensor 
data in terms of predefined "events" and, (b) drive some appropriate informa- 
tion display in real time. Manipulator control tasks can be subdivided into 
a multitude of sensory "events", and each event may have a variety of charac- 
teristic parameters. Thus, the development of fairly general purpose event- 
driven displays requires that the logic/parametric structure of the algo- 
rithms be flexible in the sense that changing control goals or subgoals can 
be accommodated by simple call-changes in the algorithms in a given control/ 
operation environment. 


The actual event display can be implemented by alternative means, the 
selection of which depends on the application environment. For event 
displays, both audio and visual display techniques are suitable. An impop 
tant consideration for selecting or designing event displays is the warning 
effect" the display can or shall impose on the operator. By definition, t e 
occurrence of a sensory event should call the operator's attention to some 
appropriate control decision or control action, without disturbing his normal 
visual attention directed toward the overall control task. Note that the 
control can require split-second decisions. Another important consideration 
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is related to the selection of the content of the display format. How much 
and what kind of detailed information the operator should be exposed to in 
addition to the “event information" within the same general frame of 
information? Too much information can be disturbing. Too little information 
can defy the purpose. The display of uncorrelated data, or the display of 
correlated data in uncorrelated form, may impose heavy cognitive load on the 
operator. 

Properly designed event--driven displays are expected to have a number of 
benefits: (a) simplify on-line control decisions; (b) reduce errors caused 

by human factors; (c) reduce perceptual/cognitive workload on human operator 
in a real-time control environment; (d) improve overall control performance 
in control situations which many times require split-second type control 
decisions. 


IV. EXAMPLES 


A. Event-Driven Proximity Displays 

Event-driven displays have been constructed for proximity sensors on 
two arms in the JPL teleoperator project, and also for a proximity sensor 
system developed at JPL and integrated with the four-claw end effector of 
JSC to be used at their shuttle manipulator training facility. 

1 , JPL Teleoperator Arms 

Both the JPL/CURV and JPL/Ames arms are equipped with four proximity 
sensors, and the event-driven display developed recently is applicable to 
both sensor systems. Although the sensor hardware is quite different on the 
two arms, the sensor display drive software is common except for the routines 
that get the data. Similarly, the event logic is common. The details of the 
computer hardware and software are described in the Appendix. 

The general format of graphics display of four proximity sensors data is 
shown in Figure 2. The display shows a view of the "bone" of a parallel jaw 
hand and four beams emanating from the hand, two from each jaw. The beam 
lengths are proportional to the sensitive length of the sensor beams. Each 
beam length is bound to 10 cm (4 in.). 

Figure 3 summarizes the proximity events together with the event logic 
and event parameters that have been implemented. In the present implementa- 
tion the parameter D is fixed at 5 cm. D is always defined parallel to and 
halfway in between the two beams which measure roll and yaw alignments, 
respectively, and relative to the line connecting the two fingertips. The 
tolerance, T, can be set by switch inputs on the computer's front panel. 
Values from 0.5 to 7.5 cm are allowed. Any combination of the four event 
logic equations may be selected to control the event success blinker. The 
success may be defined as X alignment with a tolerance, say, of 1 cm (corres- 
ponding to about 5 degrees when the hand is fully open). Or, the success may 
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be defined as Y range of 5 cm together with X alignment to within 0.5 cm 
tolerance (corresponding to about 2.5 degrees when the hand is fully open). 
This latter "success case" would be useful in moving the hand over a table 
to a wall while holding an object vertical. With this event logic, the hand 
roll angle would be small as the range measurement is made on both sensors 
and the object would be held with the hand 5 cm above the table. The final 
approach to the wall would be reached with the hand perpendicular to the wall. 

The event indicator blinker has initially been placed in the top left 
hand corner of the monitor screen. Though all four sensor beams are sho^ on 
the monitor, the operator does not have tc evaluate the four beams quantita- 
tively in terms of a predefined event. This is done for him by the display 
drive logic automatically and in real time. He can take a more qualitative 
look at the four beams to determine, e.g., why the success blinker is off ; 
that is, what to do in order to get the success blinker "on". 


Figure 4 shows two uses of the event-driven proximity display. The 
first pair of photographs (Figure 4A) shows the hand above a table and 
skewed to a block. The task is to achieve alignment with the table and the 
block. The display shows the operator how to bring the hand perpendicular 
to the block while maintaining the hand level at 5 cm above the table. The 
second pair of photographs (Figure 4B) shows that this has occurred and the 
event blinker has come on. The third pair of photographs (Figure 4C) shows 
a different alignment problem. Here, it is desired to bring the hand in 
level over the plate on the table. There are no forward references. Follow- 
ing the required corrections as indicated by the display, the desired level 
state is achieved, and the event blinker comes on as shown in the fourth pair 
of photographs (Figure 4D) . 


While the two uses of the event-driven display shown in Figure 4 are 
simple, they do demonstrate the usefulness of the concept. As more complex 
tasks are performed and analyzed, a detailed examination of the benefits can 
be made. Future improvements in implementation are also planned to enable a 
broader variety of events to be defined. Ranges, alignment angles and toler- 
ances could be individually defined rather than being commonly constrained 
as at present. 


2. JSC Four-Claw End Effector 

A proximity sensor has been developed for and Integrated with a four- 
claw end effector of JSC. The purpose of this sensor system is to aid the 
operator to find the proper final depth positioning and pitch and yaw align- 
ments of the four-claw end effector on a l6-m long manipulator relative to 
the grapple fixture of a large payload. The overall control is visually 
guided . 

The sensor system, together with the grasp envelope and measurement 
definitions are shown in Figure 5. The use of the sensor system is presently 
restricted to the verification of a "successful grasp state" before grasp 
action is initiated. The "successful grasp state" is defined by the 
dimensions of the grasp envelope (see Figure 5) and by the dynamics of grasp. 


When a successful grasp state" has been reached, the data processing 
electronics automatically turns on a simple "success display" (a buzzer or 

a green light, or both), indicating to the operator that he is ready to 
grasp * 

The data processing required to drive the "success display" has two 
modes: analog and digital. The analog drive logic implementation is quite 

simple as indicated in Figure 6. In fact, with this simple analog implemen- 
tation the full capabilities of the sensor system cannot be utilized to 
account for all physically possible combinations of depth, pitch and yaw 
error states which, due to the dimensions of the end effector's grasp enve- 
lope, still would allow successful grasp. To achieve a full utilization of 
the sensor system capabilities versus all allowable depth, pitch and yaw 
error combinations, 'success algorithms" have been developed and implemented 
us ng an Intel 80/20-4 single-board microcomputer together with an Intel 
single-board A/D converter. 

For the purpose of experimentation, several "success algorithms" have 
been implemented in the digital computer to drive the displays. The algo- 
rithms are simple, and account for all (or for almost all) allowable error 

successful grasp. Algorithms have also been imple- 
mented which utilize the outputs of any three out of the four sensors to 
indicate the success states". This is useful If one sensor eventually 
tails, or if one sensor eventually misses the top (reference) surface of the 
grapple fixture due to allowable lateral alignment errors. (Note that the 
four-sensor configuration is redundant to define and compute depth, pitch 
and yaw errors. A triangular configuration of three sensors would be 
sufficient for that purpose.) 

For the sake of brevity, only one "success algorithm" is shown in this 

mill algorithm" since it con- 

^n s the individually allowable pitch and yaw errors into a simple allow- 
able cone angle error condition. (See Condition 2 in Figure 7.) Three kinds 
of success definitions" have been developed, each with three sets of 
success parameters . All nine variations have been implemented for "all 

ftoLd^li All together 18 algorithms are 

stored in mOM in the microcomputer. Any one of the 18 algorithms are 

L callable by dialing the appropriate number between 1 and 18 on a BCD 
switch integrated with the mlcrocoinputer • 

Very successful operational ground tests have been conducted with the 
and simple display system at JSC using the 16-m long arm of the 
JSC Manipulator Development Facility in realistic large payload handling 
experiments. Fig. 8 shows a floor set-up scene (direct visual contact with 
.arget) for capturing a moving target. All together 112 test runs have been 
performed by 4 operators. The final result is that, when the "success dis- 
play .tone or green light) was on, the operators got a capture every tlme- 
There were no operator mistakes under sensor-indicated grasping conditions, 
and the sensors never indicated wrong conditions for grasping. Three of the 
four operators favored the buzzer for "success display". The utility of the 
display Increased with task difficulty. The display was required to aid the 
operator to successfully complete the most difficult tasks without error. 
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With event-driven displays, the task could be performed as follows: 
using a proximity event display set In a position sensing mode, the operator 
moves the manipulator rapidly towards the object. When signalled Chat the 
manipulator Is near the object, the operator slows Its motion and switches 
the display to a force/torque even mode. When contact has been achieved, the 
operator Is again signalled before the forces reach an unacceptable level. 
Thus, event-driven displays used In combination hold great promise for even 
greater benefits In that Casks can be performed more rapidly, more reliably 
and with lower expenditure of resources. 

C. Event-Driven Touch Display 

The touch sensor unit being used here has two 4 by 8 matrices of points 
that can sense applied pressures. These matrices, or perhaps some with higher 
point density, can be mounted on the Inside of mechanical fingers (jaws) and 
used to sense contact areas or the location of points of contact between fin- 
ger and object. Similar units could be mounted on other surfaces to sense 
other contact forces or patterns of contact areas. At each point of the sen- 
sor matrix the pressures applied locally are sensed by measuring the conduc- 
tivity of a pressure sensitive plastic. The measurement concept and the 
actual sensing elements are shown In Fig. 12. 

Fig. 12 also shows the basic touch sensor displays. The numeric 
representation of the sensor output gives a more quantitative impression of 
the applied forces distribution and Is particularly useful for diagnostic 
work. The color or B/W shades displays are more graphic and are easier to 
understand at a glance although less Information is presented. 

A particular event-driven touch sensor display Is planned to be Imple- 
mented to further enhance the control context of data presented to the 
operator. The display concept (shown in Fig. 13) is aimed to give a quanti- 
tative Indication to the operator when the contact area increases by pre- 
defined amount. While the pressures applied will still be shown as dark or 
light shades of a color, the color itself will be changed to reflect the total 
applied pressure over a given area. The matrix displayed may be red. If less 
than half the sensitive points have made contact; orange. If between 1/2 and 
3/4 have; or green. If more than 3/4 have. Thus, a green condition will 
signify a safe grasp. 


V. CONCLUSIONS 


1) Performance tests conducted at JSC with a three-dimensional proxi- 
mity sensor system and "go-no go" display have shown the basic utility of a 
simple event-driven display which conveys critical control information to 
the operator based on real-time algorithmic evaluation of multidimensional 
data. 


2) In general, event-driven displays enhance the control context of 
sensory Information since events can be defined with respect to critical 
control decisions or control actions. 
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3) Preliminary experiments strongly SngirperLpt 10 ?? oriwt . 

visual and non-visual sensory n yith sensory information over- 

This may require the development of TV monitors witn senso , 

layed to or cut into the camera information. 

4) Extensive experimentation is for- 

display formats in order to * ^ty encounter questions in human 

mats. The experimentation will by ««®® J oblective mea- 

of event-related human factors. 
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APPENDIX 


Computer Syatea 
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encoded into three 8 bit parallel words. The two computers run 

since the TV graphics display process is much slower than the ° 

bulb and tone process (about 16 times per second versus about 100 times per 

second.) 

2. Software System 

The software for the display programs has been written In 
to maximize the signal processing and display formatting «!tL! 

effective approach since the processing and formatting 

pitically simple. Typically, the symbolic language *=*''** 

10-15K bytes to store, and the machine language version l-ZK bytes. 

The programs have been written in a structured subroutine format, ^e top 
level is a sequence of calls to subroutines. The first calls ate to toutlnes 
which initialize constants and set up the displays. These are followed by the 
Iain program loop which calls routines t to see if the display program should be 
exited, change parameters based on switch or keyboard inputs, 
calibration of data, perform logic tests, format the data for the displays, . 
Each of these subroutines is a complete logical entity, so that new 
may be added by simply inserting new calls. A similar approach has been taken 
for the lower levels of subroutines. The program data structures have 
designed so that they allow an EPROM version of the programs. Thus to 
tests only a small fraction of the computer system is needed. Further the opera 
tlon of the system for demonstrations is simplified. 

The program for displaying proximity sensor data from the sensors on the 
JPWCURV arm and performing event logic is typical of the display 
grams. The first level structure is shorn in part A of Fig. 15. The actual 
process for getting the data, processing and displaying it are ® 

of Fig. 15. The modularity of the structure was a significant help in adding 
the event logic and display to the prior programs. All J** 

to add the two blocks which perform the event logic tests and which time "“a 
display the event blinker. Likewise, when changing the JPL/Ames arm proximity 
sensor program to accept data from the JPL/CliRV arm proximity senate all that 
tinecessSry was to change the "Read Sensor Data" subroutt le. The subroutines, 
incidentally, were taken from a previous force-torque sensor program. 
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EQUAL LENGTH (Y ) OF TWO BEAMS ®UAL MAGNITUDE OF TWO 

(D, AND DJ CAN BE AN "EVENr' ORTHOGONAL FORCE COMPONENTS 
‘ 2 CAN BE AN "EVENT" 


Figure 1. 


Examples for 


i 


Event Definition 



Figure 2. General Graphlce Dieplay of Proalnlty Senaor Beane 
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= RANGE MEASURED BY SENSOR '' i ", i - 1,2, 3,4 
D = DISTANCE \ p^^R/^^ETERS 

T = TOLERANCE ) 

LOGIC DEFINITIONS FOR EVENT E^- = {k| 

E^- =0FORK>0: EVENT ISN'T THERE, BLINKER "OFF" 
E^. =1FORK<0: EVENT OCCURED, BLINKER "ON" 


EVENTS 

LOGIC EQUATIONS 

1. yaw ALIGNMENT AT 

E, = 1 Is, -D| -T>0 I*! ISj -D| -T>0l 

"D" WITH "T" TOLERANCE 


2. YAW ALIGNMENT ONLY 

Ej=llS,-S2l-T>0( 

WITH"T" TOLERANCE 


3. ROLL ALIGNMENT AT 

E3-{|S3-0|-T>OH |S4-D|-T>0} 

"D" WITH "T" TOLERANCE 


4. ROLL ALIGNMENT ONLY 


WITH "T" TOLERANCE 



Figure 3. Proximity Sensing Events Example 
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Figure 5* Four Proximity Sensors on JSC Four-Claw End Effector 










- MEASUREMENTS - 



”a ”b ^*c ”d *"• 

C IS A MEASURE FOR PITCH AND YAW ERRORS; C » f(H) 


- SUCCESS LOGIC - 


WHERE H * 1/2 (D^ + Dg) 
1/2 (Dg + D^) 

© (D, -d/ + (02-d/<L 

WHERE L - - [f(H)]^ 

IF BOTH CONDITIONS ARE TRUE 
THEN LIGHT/BUZZER ARE ON, 
OTHERWISE OFF 


Ha , Hr , He , Hjj AND K 

(AND IMPLICITLY ALSO C*) 
ARE PRESET CONTANTS 


f(H) IS GIVEN BY THE 
TRAPEZOID FORMULA 
SHOWN ABOVE 


Figure 7. Conic Algorithm Indicating Acceptable Combinations of Range, Pitch | 
and Yaw Errors for Successful Grasp Using Four Proximity Sensors ; 
Integrated with JSC Four-Claw End Effector ! 
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_ . i-uo T^p Manioulfttor Develop®®^ ^ Facility Usiti® 

*“* *• Jrox^S'seHsor. Integrated with JSC Four-Claw End EEIeetor and 
Simple "Go-No Go" Event Display 
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TRANSFORMATION MATRIX 
UNDER IDEAL CONDITIONS 
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FORCES 

AND 

TORQUES 

REFERENCED 

TO 

X-T-Z 

SENSOR 

COORDINATES 


0 0 k|3 0 0 0 k^7 0 

k2| 0 0 0 k25 0 0 0 

0 l«32 0 l<34 0 1(36 0 

0 0 0 k44 0 0 0 1(46 

0 1(52 0 0 0 1(56 0 0 


FORCES 

SENSED 

AT 

SPOKE 

ELEMENTS 



Figure 10. Force-Torque Sensor Measurement Transformation 
and Graphics Display 
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Figur« 12. Multipoint Proportional Touch Sensors with Nunerlc 
and Color Graphics Displays 






INCREASE OF CONTAa AREA 


DISPLAY 
FORMAT OF 
ix8»32 
SENSITIVE 
CELLS: 



m'mm 

m'smm 

mwm. 

■■BB 


'my/WA 

'^ymWA 

'mwA 

W^aWA 

mmvA% 

WMA 




IF NO. OF SENSITIVE SC <16 
CELLS (SC) UNDER 
PRESSURE IS: 


16<$C<24 SC 5 


THEN COLOR IS: 


RED YELLOW GRi 


(NB: THE SHADED CELLS ARE THOSE UNDER PRESSURE. 
THEY HAVE TONES IN THE RESPECTIVE COLORS 
DARKER THAN THE UNSHADED CELLS.) 


Figure 13. Touch Sensing Event Example 





























